Image/information display system and method based on temporal psycho-visual modulation

ABSTRACT

An image/information display system and method based on temporal psychovisual modulation in the technical field of image/video processing and electro-optical display. The system includes a high speed display device and an auxiliary viewing device cooperating with each other and interacting with a human visual system. The auxiliary viewing device realizes the temporal amplitude modulation of the composition basic sequence by a method of controlling the light intensity transfer rate from the display plane of the high speed display device to the human eye. The present invention can not only display different pictures required individually by a plurality of users with the same display device but can also solve the problems of pictures being insufficiently bright and the number of pictures that can be generated being very low; such problems being unable to be overcome by the time division multiplexing method.

This application is the U.S. national phase of International ApplicationNo. PCT/CN2012/077353 Filed on 21 Jun. 2012 which designated the U.S.and claims priority to US Application Nos. 61/457,859, 61/457,944,61/573,060, 61/632,564 filed on 22 Jun. 2011, 14 Jul. 2011, 22 Aug. 2011and 26 Jan. 2012 respectively, the entire contents of each of which arehereby incorporated by reference.

TECHNICAL FIELD

This invention is a method and system for information display involvingthe fields of image/video signal processing and optoelectronic displays.More specifically, it is a multiview information display method andsystem using a high refresh rate display and viewing devices that carryout a computational information display process called TemporalPsychovisual Modulation (TPVM).

BACKGROUND ART

The human visual system (HVS) cannot resolve temporally rapidly changingoptical signals beyond flicker fusion frequency (around 60 Hz for mostviewers and under most conditions). But modern optoelectronic displayscan operate at much higher refresh rates, e.g. 120 Hz, 240 Hz andbeyond. For example, the new light modulators such as the deformablemirror devices and grating light valve devices can lead to very highrefresh rate (up to 88 k Hz) and spatial resolution for digitalprojectors; even inexpensive mainstream liquid crystal (LC) displays nowoffer 120 Hz or 240 Hz refresh rate, as demanded by emergingapplications of stereoscopy.

A high-speed optoelectronic display device can broadcast in visiblespectrum a far greater amount of visual stimuli than any viewer canpossibly assimilate. Thus, a single display has extra capacity, i.e.,psychovisual redundancy, to generate multiple visual percepts for aplurality of users. A straightforward way of exploiting psychovisualredundancy of high-speed displays is time multiplexing. Sony's dual-viewtechnology presents two game participants their respective views on thesame screen but in disjoint time slots (US2010/0177172A1, publicationdate 2010-07-15, “Stereoscopic screen sharing method and apparatus”;US2010/0177174, publication date 2010-07-15, “3D shutter glasses withmode switching based on orientation to display device”). Kulik et al.proposed a 360 Hz display system to generate six stereoscopic viewsbased on time multiplexing and polarization (A. Kulik, A. Kunert, S.Beck, R. Reichel, R. Blach, A. Zink, and B. F. Yoehlich, “A stereoscopicsix-user display for co-located collaboration in shared virtualenvironments”, ACM Transactions on Graphics, vol. 30, no. 6, 2011). Buttime multiplexing is a very inefficient approach of utilizingpsychovisual redundancy of a high-speed display, because it completelyignores statistical redundancy among the output images. As a result, thenumber of different views K can only increase linearly in the displayspeed; moreover, the light influx of each view also decreases linearlyin K. These constraints greatly limit the number of concurrent viewsthat can be produced by time multiplexing. Another major drawback ofmultiplexing type of multiview display methods is that they cannotgenerate meaningful images for viewers who do not use any auxiliaryviewing device.

DISCLOSURE OF THE INVENTION

This invention is a fundamental departure from time multiplexing; itovercomes the above mentioned shortcomings of time multiplexing. The keyinnovation is to fully exploit both psychovisual redundancy of thehigh-speed display devices and statistical redundancy of visual signalsby temporal amplitude modulation of two-dimensional signals in visiblespectrum. In the invented multiuser display method and system, ahigh-speed display or projector sequentially emits a set of so-calledatom frames, which are a two-dimensional grid of pixels. Unlike in timemultiplexing, these atom frames are not necessarily completely formedimages but rather constituent parts of a group of images to be displayedconcurrently with the same display device. These atom frames areamplitude modulated by display-synchronized viewing devices, whicheffectively apply different weights to individual atom frames. Differentusers perceive their intended images all on the same physical displayplane, as the results of their individual HVS's fusing appropriatelyweighed consecutive atom frames. Not only this invention can generate alarge number of concurrent views required individually by a plurality ofusers with a single display device, but also it can prevent the problemof low light influx for each of the concurrent views, which are notachievable by time multiplexing-based multiuser display methods.

The invented method and system of information display comprise, asdepicted in FIG. 1, a high-speed display device and accompanied viewingdevice(s) that interact(s) with the display device and the human visualsystem. The said viewing device(s) is(are) capable of adjusting, in acontinuous range, the transmittance rate of light from the display planeof the said display device to eye(s) of the viewer who uses the viewingdevice, thus performing temporal amplitude modulation of the atom framesemitted by the display device.

When the said multiview display system serves a plurality of users,their viewing devices can independently adjust the amount of lightattenuation and vary the light transmittance rate from the display planeto users' eyes in the range between 0 and 1; namely, anywhere in betweencomplete blockage to complete passage of light. Through synchronizationwith the display device that emits atom frames, each user'sindependently adjustable light attenuation viewing device carries outthe temporal amplitude modulation of atom frames that is required toform the image for the user in his/her own human visual system. Becausethe same sequence of atom frames can be amplitude modulated differentlyby different users' viewing devices, a single high-speed display devicecan concurrently exhibit a plurality of images without interference. Theabove described methodology and system represent a new paradigm ofcomputational information display, called temporal psychovisualmodulation (TPVM). The display-synchronized light attenuation viewingdevices are referred to as TPVM viewing devices in the sequel. Thesynchronization between the high-speed display device and the TPVMviewing devices can be achieved readily by any one of many low-bandwidthshort-range communication methods, wired or wireless. The method ofcomputing temporal amplitude modulation patterns for different userswill be described in detail shortly in this document.

One or more users can use their TPVM viewing devices to carry out aparticular temporal light amplitude modulation so that he can seeprivate/secret information or image on a display plane that areinvisible or illegible to those who do not use the light-modulationviewing device. Similarly, at the same time when the TPVM display systemgenerates different concurrent images to be perceived by users who usetheir TPVM viewing devices, it can also display yet another meaningfulimage to those who do not use any TPVM viewing device at all.

By a slight generalization, the display-synchronized TPVM viewingdevices can perform different light amplitude modulations on differentgroups of pixels or on different pixels.

Also, different light amplitude modulations can be applied to the leftand right eyes in case the TPVM viewing device has separate lens for thetwo eyes, like active LC glasses.

Next presented are the details of the TPVM computational informationdisplay method. In the following descriptions, it is assumed that thebrightness of the display device is linear in pixel value. For displaydevices not satisfying the said linearity assumption, a remapping ofpixel values to brightness levels, such as gamma correction, should becarried out to ensure the validity of the methods to be describedhereafter. Let x₁, x₂, . . . , x_(M) be the sequence of atom framesemitted by the high-speed display device of the system, where M is apositive integer. These M atom frames are amplitude modulated to form Ktarget images y₁, y₂, . . . , y_(K), where K is a positive integer.Target image y_(k), 1≦k≦K, is the result of modulating atom frames x₁,x₂, . . . , x_(M) with modulation weighting vector w_(k)=(w_(k1),w_(k2), . . . , w_(kM)) namely, y_(k)=w_(k1)x₁+w_(k1)x₂+ . . .+w_(km)x_(M), where w is the light transmittance rate of atom framex_(M), 1≦m≦M, to generate target image y_(k). The K modulation weightingvectors need to be computed and transmitted to the corresponding Kdisplay-synchronized TPVM viewing devices, one for each target image.

The K target images y₁, y₂, . . . , y_(K) constitute an N×K matrix Ywith column k being image y_(k) of N pixels; the M atom frames x₁, x₂, .. . , x_(M) constitute an N×M matrix X with column k being atom framex_(k) of N pixels; The task of the TPVM display system is to solve theproblem of signal decomposition Y=XW, where W is an M×K modulationmatrix whose columns are the K weighting vectors w₁, w₂, . . . , w_(K).In other words, computing appropriate M atom frames and corresponding Kmodulation vectors to concurrently display K given target imagesrequires the factorization of matrix Y (the K target images) into theproduct of matrix X (the M atom frames) and matrix W (the correspondingK modulation vectors). Since the display device cannot emit negativelight energy and TPVM viewing devices cannot implement negative lighttransmittance rate, the signal decomposition Y=XW has to be anon-negative matrix factorization (NMF). Moreover, the elements ofmatrix X need to be set in the normalized interval [0,1] in order toaccount for the fact that the dynamic range of the display device isfinite. Considering the current state of the art that light modulationdevices, such as active LC glasses, can only perform light attenuationnot magnification, the elements of matrix W should be constrained withinthe interval[0,1]. Therefore, given K target images, the TPVM multiviewdisplay system solves the following NMF problem

$\begin{matrix}{{\min\limits_{X,W}\mspace{14mu} {{{sY} - {XW}}}_{F}}{{{{subject}\mspace{14mu} {to}\mspace{14mu} 0} \leq W \leq 1},{0 \leq X \leq 1}}} & (1)\end{matrix}$

to obtain the M atom frames in X to drive the high-speed display device,and obtain the K modulation weighting vectors in matrix W that arecommunicated to and drive the display-synchronized TPVM viewing devices,with the objective of minimizing the errors in the reconstructed Ktarget images. The scaling factor s, 1≦s≦M, is optional to ensureadequate intensities of the concurrently displayed images.

Should more sophisticated light modulation devices become available inthe future that can perform light magnification, the upper bound on Wcan be accordingly relaxed within the scope of this invention.

The above described method is an ideal but only one of many possiblemethods of computing the atom frames and modulations weights of the TPVMmultiview display system.

Other more practical, approximate methods can be derived by skilledpersons in the field but within the scope and principle of thisinvention, and hence should be considered merely variants of the abovedescribed method and should fall within the scope of the invention.

The TPVM multiview display system transmits the modulation weightingvectors w₁, w₂, . . . , w_(K) computed as described above, possiblytogether with frame synchronization markers, to users'display-synchronized TPVM viewing devices.

Users can choose which of the K target images to perceive by receivingand applying to his/her TPVM viewing device the corresponding amplitudemodulation weighting vector; the image perceived through a viewingdevice that performs amplitude modulation according to w_(k)=(w_(k1),w_(k2), . . . , w_(kM)) is target image y_(k)=w_(k1)x₁+w_(k1)x₂+ . . .+w_(km)x_(M), 1≦k≦K, which is a linear combination of the M atom framescomputed as described above.

Physically, w_(k1), w_(k2), . . . , w_(kM) are frame-synchronizedsignals by which the TPVM viewing device controls the transmittancerates of the individual atom frame.

In the TPVM multiuser display system, a viewing device of separate lensfor the left and right eyes, such as a pair of active LC glasses, canapply different amplitude modulation vectors to the two lenses andgenerate different left-eye and right-eye images for the user, if sodesired.

Preprocessing is an effective approach to greatly reduce the complexityof the TPVM multiview display system. In practice, the atom frames andthe amplitude modulation weighting vectors corresponding to a given setof target images can be pre-computed and stored, so that they can belater fetched to respectively drive the display device and the TPVMviewing devices in real-time sessions.

The TPVM multiview display system of this invention can work inconjunction with eye-tracking or/and location tracking devices for oneor several users. Based on the present locations of users' eyes orbodies generated by these tracking devices, the TPVM multiview displaysystem computes and updates target images according to users' eyepositions and viewing angles. The resulting target images are thendecomposed into a set of atom frames and the corresponding amplitudemodulation vectors by non-negative matrix factorization; the atom framesand modulation vectors are then used to respectively drive the displaydevice and the display-synchronized TPVM viewing devices as describedabove.

Among all concurrent views generated by the TPVM multiview displaysystem, there is a particular view, called normal view. The normal viewis the result of the human visual system fusing all atom frames withoutany attenuation. In other words, the normal view is our daily visualexperience without tampering the lights from the display, which is theimage y₀=x₁+x₂+ . . . +x_(M), where M is the number of atom frames. Incontrast, other views perceived through unequal attenuations of the atomframes are called shale views, namely images y_(k), 1≦k≦K in thenotation above.

The TPVM multiview display system can choose the atom frames x₁, x₂, . .. , x_(M) and the amplitude weighting vectors w₁, w₂, . . . , w_(K)using the methods to be detailed below, so that the normal viewy₀=x₁+x₂+ . . . +x_(M) is semantically meaningful or/and visuallypleasing for those using no TPVM viewing devices; at the same time,others using TPVM viewing devices perceive shale views individuallyrequired by them.

One of the methods to concurrently display a given normal view y₀ and Kgiven shale views {y_(k)}1≦k≦K, is to solve the following equation withacceptable accuracy

$\begin{matrix}{\left( {y_{0},y_{1},\ldots \mspace{14mu},y_{K}} \right) \approx {\frac{1}{C(M)}\left( {x_{1},x_{2},\ldots \mspace{14mu},x_{M}} \right)\begin{pmatrix}1 & w_{11} & \ldots & w_{1\; K} \\1 & w_{21} & \ldots & w_{2\; K} \\\vdots & \vdots & \ddots & \vdots \\1 & w_{M\; 1} & \ldots & w_{M\; K}\end{pmatrix}}} & (2)\end{matrix}$

where C(M) is a constant depending on M used to adjust the brightness ofthe shale views and the normal view, and obtain the atom frames x₁, x₂,. . . , x_(M) and amplitude modulation vectors w_(k)=(w_(k1), w_(k2), .. . , w_(kM)) 1≦k≦K, which are required to generate the normal view y₀and the K shale views y_(k), 1≦k≦K. The capability to concurrentlyexhibit one useful or/and pleasing normal view to viewers who do not useany TPVM viewing device and one or many shale views required by userswho use TPVM viewing devices, all on a single high-speed display device,is a unique and landmark property of this invention. This property opensup many new applications that are not possible with the existinginformation display technologies.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates the principle of computational information display ofthe TPVM invention. A high refresh rate display device broadcasts asequence of atom frames; these atom frames, after being amplitudemodulated by the display-synchronized active liquid crystal glasses, arefused and perceived by the human visual system as the target image.

FIG. 2 is a schematic illustration of the TPVM multiview display systemthat exhibits to a plurality of users individually required imagesconcurrently on the same display plane, as detailed in embodiment 1.

FIG. 3 illustrates a way a display-synchronized TPVM viewing device,such as those made of active LC class, carries out different amplitudemodulations in different parts of the atom frames by spatially varyingthe amount of light attenuation, when driven by corresponding spatiallyvarying modulation weights. Individual users can have different visualexperiences independent of others' from the same display plane, viatheir own TPVM viewing devices that perform different spatially varyingamplitude modulations of the same set of atom frames. The depictedexample is that the TPVM multiview display system allows multiple usersto independently explore a 3D medical data set via their individual TPVMviewing devices of spatially varying modulation weights.

FIG. 4 illustrates the method and system of security display based onthe bifurcation between the normal view and shale view, as detailed inembodiment 3.

FIG. 5 illustrates the method and system of multilingual presentationsof content on the same display plane, as detailed in embodiment 3.

FIG. 6 illustrates the method and system of backward-compatiblestereoscopic display by a joint construction of a 2D normal view forviewers using no 3D glasses and two shale views, one per eye, forviewers using 3D glasses, as detailed in embodiment 2.

BEST MODE FOR CARRYING OUT THE INVENTION Embodiment 1 Multiuser VirtualReality and Augmented Reality

As illustrated in FIG. 2, the TPVM multiview display system can exhibitto a plurality of users individually required images concurrently on thesame display plane. The high-speed display device broadcasts to allusers a common set of atom frames. Through synchronization with theabove said display device, each user's independently adjustablelight-attenuation viewing device carries out an appropriate temporalamplitude modulation of the atom frames to form the image for the userin his/her own human visual system.

The unique ability of the TPVM multiview display system to generatedifferent concurrent interference-free views on the same physicaldisplay plane adds a valuable social dimension to applications ofvirtual reality (VR), augmented reality (AR) and visualization. Thisinvention can provide, in conjunction with eye trackers and/or motionsensors, a plurality of users with their own perspective-correct views;for instance, in a virtual stadium, users can watch the same ball gamein their own vantage points while enjoying each other's accompany. Inthis setting, exchanges on whether a ball crosses the goal line, or thealike, become more interesting and engaging.

This invention can provide an intellectually stimulating and productiveplatform for collaborative investigation and learning in VR or AR. Forexample, when a group of physicians meet face to face to visuallyexamine a difficult case using a TPVM multiview display system, everyonecan independently explore the 3D anatomy of the patient in his/her ownperspective while pointing to the object of interest on screen andexchanging thoughts with colleagues. Such a group experience ofexploring a virtual environment in physical co-presence is eitherimpossible or unnatural with current display devices of VR and AR,because head mounted displays hinder face-to-face communication.

Moreover, these personal AR/VR display devices like Google Glass do notallow multiple users to work with the same display medium. In the TPVMmultiview display system, a display-synchronized viewing device does nothave to perform the same amplitude modulation everywhere in an atomframe. As shown in FIG. 3, a TPVM viewing device, such as a visor madeof active LC class, can carry out different amplitude modulations indifferent parts of the atom frames by spatially varying the amount oflight attenuation, when being driven by corresponding spatially varyingmodulation weights. By spatially varying amplitude modulation weights ofthe display-synchronized TPVM viewing device, the TPVM multiview displaysystem can create many new interesting multi-user visual experiences.FIG. 3 shows such an example for 3D medical visualization. Specifically,multiple layers of registered human anatomy images are mapped to atomframes: atom frame 1 is the outer human body; atom frame 2 is theskeleton; atom frame 3 is the inner organs. These atom frames arecyclically exhibited at high speed to users, each of whom watches thedisplay through his/her own display-synchronized active LC visor (anembodiment of TPVM viewing device in this case). The area of thedisplay-synchronized LC visor is partitioned into inner and outer parts.In the inner part the M atom frames are amplitude modulated according tomodulation weighting vector w_(i)=(w_(i1), w_(i2), . . . ,w_(iK))ε[0,1]^(K); in the outer part the M atom frames are amplitudemodulated according to modulation weighting vector w_(o)=(w_(o1),w_(o2), . . . , w_(oK))ε[0,1]^(K).

By choosing different modulation weighting vectors w_(i) and w_(o) ofhis/her TPVM visor, a user can experience different desired see-throughvisual effects of the human body. For instance, by setting w_(i)=(0,0,1)and w_(o)=(1,0,0), a user sees a selected internal organ in alignmentwith the exterior of the human body. It is also straightforward toperform Alpha blending of anatomy layers by adjusting the modulationweights of the display-synchronized LC visor. Similarly, spatiallyvarying TPVM modulation weighting vectors can be used to visualize a raw3D volume data set in various ways. For example, 2D sections of a 3Dvolume, denoted by x₁, x₂, . . . , x_(M), are cyclically displayed athigh speed as atom frames. The display-synchronized LC visor changes theamount of light attenuation in a concentric pattern under the control ofthe corresponding spatially varying modulation weighting vector. In thisway, the TPVM multiview display system can present to multiple usersdifferent funnel-shaped “dig-in” views into the 3D volume (mosaic ofdepth layers) concurrently on the same physical display plane.

When being used in multiuser VR/AR applications, the TPVM multiviewdisplay system has a distinct advantage over user-wearable displaydevices such as head-amounted displays and Google Glass in that it movesheavy computations of image formation from personal devices anddistribute them to a powerful server that solves the underlying TPVMproblem of non-negative matrix factorization, to a high-speed display,and to the user's own HVS. As a result, the terminal user AR/VR devicebecomes much simpler, lighter, far less expensive and energy efficientthan existing ones. Moreover, the TPVM multiview display paradigm cangreatly reduce the computation power and video memory bandwidthrequirements of real-time VR/AR applications, because a small number ofatom frames, which can be pre-computed for a given virtual environment,can synthesize a large range of different views through appropriateamplitude modulations of the pre-computed atom frames that are performedby users' viewing devices. The communication bandwidth for transmittingTPVM modulation vectors to display-synchronized viewing devices isnegligibly small in contrast to the bandwidth required to transmitimages/videos to end user devices.

Embodiment 2 Backward-Compatible Stereoscopic Display

As shown in FIG. 6 and as detailed in the description of the invention,by properly choosing atom frames x₁, x₂, . . . , x_(M) and the amplitudeweighting vectors w₁, w₂, . . . , w_(K), the TPVM multiview displaysystem can simultaneously exhibit one or many views to one or manyuser(s) who use display-synchronized viewing device(s) and a normal viewy₀=x₁+x₂+ . . . +x_(M) that is meant for those who use no TPVM viewingdevices, all on the same physical display plane. The utility of theabove said TPVM functionality can be exemplified by another embodimentof this invention: backward-compatible stereoscopic display system. Herethe backward compatibility of a stereoscopic display system means thaton the same physical display medium a three-dimensional image and atwo-dimensional image of a scene can be exhibited concurrently; viewerswearing stereoscopic glasses perceive the three-dimensional image, whileviewers wearing no stereoscopic glasses perceive the two-dimensionalimage. This invention can realize a backward-compatible stereoscopicdisplay system, by creating K=2 with shale views y_(L) and y_(R), whichare the left-eye and right-eye images of a stereoscopic scenerespectively, plus a normal view y₀ that is a two-dimensionalrepresentation of the same stereoscopic scene. Without loss ofgenerality, an exemplary backward-compatible stereoscopic display methodand system are described below.

Given a stereoscopic image (y_(L), y_(R)) and an accompanytwo-dimensional image y₀, M atom frames x₁, x₂, . . . , x_(M) and twoamplitude modulation weighting vectors w_(L)=(w_(L1), w_(L2), . . .w_(LM)) and w_(R)=(w_(R1), w_(R2), . . . w_(RM)) are jointly computedvia the following non-negative matrix factorization:

$\begin{matrix}{\left( {y_{0},y_{L},y_{R}} \right) \approx {\frac{1}{C(M)}\left( {x_{1},x_{2},\ldots \mspace{14mu},x_{M}} \right)\begin{pmatrix}1 & w_{L\; 1} & \ldots & w_{R\; 1} \\1 & w_{L\; 2} & \ldots & w_{R\; 2} \\\vdots & \vdots & \ddots & \vdots \\1 & w_{LM} & \ldots & w_{RM}\end{pmatrix}}} & (3)\end{matrix}$

where C(M) is a constant dependent on M. The left lens of thestereoscopic glasses amplitude modulates the resulting M atom frames x₁,x₂, . . . , x_(M) with modulation weighting vector w_(L)=(w_(L1),w_(L2), . . . w_(LM)); the right lens of the stereoscopic glassesamplitude modulates the same set of atom frames with modulationweighting vector w_(R)=(w_(R1), w_(R2), . . . w_(RM)). Therefore,viewers wearing the said TPVM stereoscopic glasses perceive thethree-dimensional image (y_(L), y_(R)) as the results of themodulations. On the same display medium and at the same time, viewerswearing no stereoscopic glasses perceive the two-dimensional image y₀,which is the result of directly fusing the atom frames x₁, x₂, . . . ,x_(M) by the human visual system.

Other computationally more efficient implementations of the TPVM-basedbackward-compatible stereoscopic display method and system are alsopossible. For example, the concurrent 3D and 2D views can be generatedby finding the solutions of the following linear least-square problem

$\begin{matrix}{{\min\limits_{x_{1},x_{2},\ldots \mspace{14mu},x_{M},\alpha}\mspace{14mu} \left\{ {{{x_{1} - y_{L}}} + {{x_{M/2} - y_{R}}} + {\lambda \cdot {{{\alpha \; y_{0}} - x_{1} - x_{2} - \ldots - x_{M}}}}} \right\}}\mspace{20mu} {{{{subject}\mspace{14mu} {to}\mspace{14mu} 0} \leq x_{1}},x_{2},\ldots \mspace{14mu},{{x_{M} \leq 1};{1 < \alpha \leq M}}}} & (4)\end{matrix}$

where the Lagrangian multiplier 2 is used to balance the visualqualities of the resulting 2D and 3D views; coefficient α determinesoverall intensity of the 2D image and it is adjusted in the optimizationprocess to eliminate or suppress artifacts of the 2D image.

Yet an even simpler, real-time approximate method of choosing the atomframes for the backward-compatible stereoscopic display is to directlyset the left-eye image y_(L) and the right-eye image y_(R) to two of theatom frames, say x_(i) and x_(j), 1≦i, j≦M, M being the number of atomframes, and make the 2D image y₀ be either the left-eye image y_(L) orthe right-eye image y_(R). The other atom frames x_(t) t≠i,j, are chosensome way such that x₁+x₂+ . . . +x_(M)≈y₀. For instance, for M=4, apossible method is

x ₁ =y _(L) ,x ₂ =αy ₀ −y _(L) ,x ₃ =y _(R) ,x ₄ =αy ₀ −y _(R)  (5)

the resulting atom frames x₁, x₂, x₃, x₄ are displayed cyclically athigh speed and are amplitude modulated by the left and right active LCeye glasses according to modulation weighting vectors w_(L)=(1,0,0,0)and w_(R)=(0,0,1,0), respectively. This process generates a stereoscopicview for all users who wear the above said 3D glasses. At the same timeand on the same TPVM display medium, viewers without 3D glasses perceivea 2D view y₀=x₁+x₂+x₃+x₄ of the same scene.

In the above described TPVM-based backward-compatible stereoscopicdisplay system, the choice of the 2D image y₀ to be synthesized by TPVMfor viewers wearing no 3D glasses may vary in time; y₀ can be made toswitch from the left-eye view y_(L) to the right-eye view y_(R) or toany other in-between view, back and forth. This 2D view y₀ should bechosen to minimize the overall amount of perceptual artifacts. In orderto prevent jittering in the synthesized 2D video, the switch of the 2Dviews should coincide in time with scene changes in the video.

While the foregoing written description of the TPVM-basedbackwards-compatible stereoscopic display/projection method and systemenables one of ordinary skill to make and use what is consideredpresently to be the best mode thereof, those of ordinary skill willunderstand and appreciate the existence of variations, combinations, andequivalents of the specific embodiment, method, and examples herein. Theinvention should therefore not be limited by the above describedembodiment, method, and examples, but by all embodiments and methodswithin the scope and spirit of the invention.

Embodiment 3 Security Display, Multilingual Display and the Alike

In the TPVM multiview display method and system, the normal view y₀formed without amplitude modulation of atom frames and a shale viewy_(k) formed with amplitude modulation of atom frames can be madevisually and semantically very different.

By deliberately making the normal view y₀ unrelated to the shale viewy_(k), the TPVM display method and system can use the normal view y₀ asa camouflage of a secret image that is the shale view y_(k). Like thecover text in steganography, the normal view y₀ misleads unaided eyeswhereas the shale view y_(k) carries secret messages that are onlyreadable through a display-synchronized TPVM viewing device, which canbe encrypted through a time varying modulation weighting vectors. Thesecurity display method uses a subset of atom frames to generate thesecret message and it uses another subset of atom frames to generate thecamouflage or interference visual signals. When all the atom frames aredisplayed at high speed, only authorized user(s) who are equipped withthe above said TPVM viewing device(s) can read the secret message; allother viewers will see a decoy image or an inference image (e.g., whitenoises). The above described TPVM-based security display method allows auser to work with confidential information on a personal device inpublic areas with no worry of being eavesread or peeped as depicted inFIG. 4. This invention offers much stronger protection than opticalprivacy film of the 3M company that can only block certain viewingangles and cannot prevent eavesreading or peeping from behind. Ifimplemented on a touch screen of high refresh rate, this invention canrealize a privacy keypad that only the user can operate via a specialviewport or glasses. The TPVM-based privacy keypads can be used onautomatic teller machines, electronic payment machines, combinationlocks and etc.

There are many embodiments of the above described TPVM-based securitydisplay method and system. What is described below is only one possibleembodiment. Suppose that a display device can emit M atom frames withoutcausing objectionable flickering. Given a secret image y₁ for authorizedusers who use TPVM viewing devices and a cover (decoy) image y₀ intendedfor unassisted eyes, the perceptual bifurcation effect for the purposeof security/privacy display can be achieved by computing the M atomframes x₁, x₂, . . . , x_(M) and the amplitude modulation vector w=(w₁,w₂, . . . , w_(M)) to satisfy the following equation sufficiently well

$\begin{matrix}{\left( {y_{0},y_{1}} \right) \approx {\frac{1}{C(M)}\left( {x_{1},x_{2},\ldots \mspace{14mu},x_{M}} \right)\begin{pmatrix}1 & w_{1} \\1 & w_{2} \\\vdots & \vdots \\1 & w_{M}\end{pmatrix}}} & (6)\end{matrix}$

where C(M) is a function of M. The resulting M atom frames x₁, x₂, . . ., x_(M) are displayed cyclically at high speed, and they generate coverimage y₀ for those without using matched TPVM viewing devices and at thesame time they generate secret image y₁ for the authorized user(s) whois equipped with a matched, possibly encrypted, viewing device. The TPVMviewing device is driven by the modulation vector w=(w₁, w₂, . . . ,w_(M)) while being in synchronization with the TPVM security display.

More cost-effective embodiments of the TPVM-based perceptual bifurcationmethod exist. For example, a 120 Hz display can generate cover image y0and flicker-free secret image y₁ simultaneously using M=2 atom framesx₁, x₂. The best legibility of the secret message, when being readthrough the TPVM viewing device, requires x₁=y₁, or equivalently themodulation vector w=(w₁, w₂)=(1,0). With x₁ and w given, the TPVM-basedperceptual bifurcation method chooses the remaining atom frame x₂ suchthat the resulting cover image y₀=x₁+x₂ can best camouflage secret imagey₁. The simplest way is to make the normal view y₀ a random noise imagen by setting x₂=n−x₁.

Within the scope of this invention there is another class ofapplications, in which the normal view y₀ is not meant to conceal as insecurity display applications but rather to be a default view formajority or casual viewers; on the other hand, one or multiple shaleviews y_(k), 1≦k≦K, K being a positive integer, are for some user(s) ofspecial needs who have to share the physical display medium with others.For example, often a public speaker (e.g., teacher, entertainer,politician, etc.) desires to follow but without appearing to read notesduring his/her slides presentation. This invention allows private notesto be projected onto the screen that are transparent to the audience butvisible to the speaker only. In this case, the normal (default) view isthe visual intended for the audience, whereas the shale view, which isan annotated normal view, can be seen by the presenter via a TPVMviewing device (e.g., active LC modulation classes). Similarly, the TPVMdisplay method and system can support concurrent multilingual visualpresentations on the same physical display medium, which is a visualequivalent to the oral form of simultaneous interpretation. In thiscase, as illustrated in FIG. 5, the normal (default) view displays thecontents in an international language, say English, which can be readwithout using any TPVM viewing device; at the same time, the K shaleviews, y_(k), 1≦k≦K, display the same contents but in other languages(e.g., Chinese, French, Japanese, Korean, etc.), which can be read usingcorresponding TPVM viewing devices.

This invention can also be used to build novel multiuser display systemsfor computer or electronic gaming. For example, two game participantscan play against each other on the game console using their own TPVMviewing devices (e.g., active LC modulation glasses); player 1 receivesvisual percept y₁ meant for this player but not the visual percept y₂meant for player 2, and vice versa, although they both share the samephysical display medium. At the same time, all observers of the game canhave visual percept y₀ that is meaningful and interesting for themrather than a cluttered, confusing superimposition of y₁ and y₂.

What is claimed is:
 1. An image/information display system based onTemporal Psychovisual Modulation (TPVM), comprising: a high-speeddisplay device that sequentially emits a sequence of atom frames, andaccompanied TPVM viewing device(s) which interact(s) with the displaydevice and the human visual system, the said viewing device(s) is(are)capable of adjusting, in a continuous range, the transmittance rate oflight from the display plane of the said display device to the eye(s) ofthe viewer who uses the TPVM viewing device, thus performing amplitudemodulation of the temporal sequence of atom frames displayed by the saidhigh-speed display device; moreover, the said temporal sequence of atomframes, if not being modulated, can be directly fused by the humanvisual system into a clear, meaningful image.
 2. The system as claimedin claim 1 wherein a plurality of users can use their TPVM viewingdevices to perform amplitude modulations of the temporal sequence ofatom frames displayed by the said high-speed display device individuallyso that they perceive different target images, called shale views; theshale views are in general different from the so-called normal viewperceived by those who do not use TPVM viewing devices and thus see theresult of direct fusion of all atom frames.
 3. The system as claimed inclaim 1 wherein the said viewing device(s) have the option of performingdifferent temporal amplitude modulations on different groups of pixelsor on different pixels of atom frames independently, allowing spatiallyvarying amplitude modulations.
 4. The system as claimed in claim 1 orclaim 2 or claim 3 wherein the said viewing device performs temporalamplitude modulation of the above said sequence of atom frames byadjusting the light transmittance rate between the high-speed displaydevice and the eye(s) of the user in the range between 0 and 1; namely,anywhere in between complete blockage to complete passage of light. 5.The system as claimed in claim 1 or claim 2 or claim 3 or claim 4wherein the said viewing device is a light adjustment device, whichsynchronizes with the said high-speed display device with wire orwirelessly and receives modulation weighting vector as control signal,thus adjusting the transmittance rate of light between the said displaydevice and the eye(s) of the viewer.
 6. The system as claimed in claim 1or claim 2 or claim 3 or claim 4 wherein the said viewing deviceperforms amplitude modulations of the above said temporal sequence ofatom frames for the left and right eye individually.
 7. The system asclaimed in claim 1 or claim 2 wherein the said temporal sequence of atomframes is modulated independently according to two amplitude modulationvectors to generate two images of a stereoscopic pair for viewer(s) withTPVM viewing device, and the above sequence of atom frames is so chosenthat it can be directly fused by the human visual system into a cleartwo-dimensional image for viewer(s) without TPVM viewing device.
 8. Thesystem as claimed in claim 1 or claim 2 wherein the said temporalsequence of atom frames is modulated according to an amplitudemodulation vector to generate an image containing secret message(s) thatis visible to authorized viewer(s) with TPVM viewing device, and theabove sequence of atom frames is so chosen that it can be directly fusedby the human visual system into another image without secret message(s)for viewers without TPVM viewing devices.
 9. The system as claimed inany of the above claims further comprising eye-tracking or/and locationtracking device(s) coupled to the said viewing device(s), and thetracking device(s) communicate and forwarding real-time information ofthe viewers' eye or/and location to the said high-speed display device,thus enables the display device adjusting the sequence(s) of atom framesand its respective modulation weighting according to each user's viewpoint and angle.
 10. The system as claimed in any of the above claimswherein the temporal sequence of atom frames and the amplitudemodulation weighting vectors that are required to generate a given setof target images are computed and stored in advance for later use, thusreducing the computational complexity of TPVM.
 11. An image/informationdisplay method based on Temporal Psychovisual Modulation comprising:displaying a sequence of atom frames x₁, x₂, . . . , x_(M), where M is apositive integer, sequentially in time, which are amplitude modulated byTPVM viewing devices independently and subsequently fused by users'human visual systems into individually required images (shale views), ordirectly fused without any modulation by the human visual system into anormal view.
 12. The method as claimed in claim 11 wherein the saidframes are presented by: different series of weighting modulating onesame sequence of atom frames, or without modulating.
 13. The method asclaimed in claim 11 wherein the said sequence of atom frames x₁, x₂, . .. , x_(M) is used to present K target images y₁, y₂, . . . , y_(K),where each target image, y_(k), y_(k)=w_(k1)x₁+w_(k2)x₂+ . . .+w_(kM)x_(M), 1≦k≦K, is modulated by a serial of weighting w=(w_(k1),w_(k2), . . . w_(kM)) on the sequence and the viewer's eye.
 14. Themethod as claimed in claim 13 wherein the K individually required targetimages y₁, y₂, . . . , y_(K) constitute an N×K matrix Y with column kbeing image y_(k) of N pixels; the M atom frames x₁, x₂, . . . , x_(M)constitute an N×M matrix X with column k being atom frame x_k of Npixels, and thus Y=XW, where W is an M×K modulation matrix whose columnsare the K weighting vectors w₁, w₂, . . . , w_(K); therefore, given Ktarget images, the required M atom frames and K weighting vector scan beobtained by solving the following NMF problem$\min\limits_{X,W}\mspace{14mu} {{{sY} - {XW}}}_{F}$subject  to  0 ≤ W ≤ 1, 0 ≤ X ≤ 1, where s is an optional scalingfactor, 1≦s≦M, to ensure adequate intensities of the concurrentlydisplayed images.
 15. The method as claimed in claim 12 or claim 13wherein the modulation weighting vectors w₁, w₂, . . . , w_(K) and thesequence of atom frames x₁, x₂, . . . , x_(M) meet the followingcondition:${\left( {y_{0},y_{1},\ldots \mspace{14mu},y_{K}} \right) \approx {\frac{1}{C(M)}\left( {x_{1},x_{2},\ldots \mspace{14mu},x_{M}} \right)\begin{pmatrix}1 & w_{11} & \ldots & w_{1\; K} \\1 & w_{21} & \ldots & w_{2\; K} \\\vdots & \vdots & \ddots & \vdots \\1 & w_{M\; 1} & \ldots & w_{M\; K}\end{pmatrix}}},$ where C(M) is a constant depending on M used to adjustthe brightness of the shale views and the normal view, thus presentingrequired target images y_(k), 1≦k≦K, to respective viewer(s) with TPVMviewing device(s) while presenting a different meaningful imagey₀=x₁+x₂+ . . . +x_(M) to those viewer(s) without using TPVM viewingdevices.
 16. The method as claimed in claim 15 wherein given astereoscopic image (y_(L), y_(R)) and an accompany two-dimensional imagey₀, M atom frames x₁, x₂, . . . , x_(M) and two amplitude modulationweighting vectors w_(L)=(w_(L1), w_(L2), w_(LM)) and w_(R)=(w_(R1),w_(R2), . . . w_(RM)) are jointly computed to approximately satisfy thefollowing${\left( {y_{0},y_{L},y_{R}} \right) \approx {\frac{1}{C(M)}\left( {x_{1},x_{2},\ldots \mspace{14mu},x_{M}} \right)\begin{pmatrix}1 & w_{L\; 1} & \ldots & w_{R\; 1} \\1 & w_{L\; 2} & \ldots & w_{R\; 2} \\\vdots & \vdots & \ddots & \vdots \\1 & w_{LM} & \ldots & w_{RM}\end{pmatrix}}},$ where C(M) is a constant dependent on M, thusviewer(s) with TPVM viewing device(s) perceive the three-dimensionalimage (y_(L), y_(R)) as the results of the amplitude modulations whileviewer(s) without viewing device(s) perceive the two-dimensional imagey₀.
 17. The method as claimed in claim 16 wherein the said sequence ofatom frames x₁, x₂, . . . , x_(M) is obtained by solving the followinglinear least-square problem:$\min\limits_{x_{1},x_{2},\ldots \mspace{14mu},x_{M},\alpha}\mspace{14mu} \left\{ {{{x_{1} - y_{L}}} + {{x_{M/2} - y_{R}}} + {\lambda \cdot {{{\alpha \; y_{0}} - x_{1} - x_{2} - \ldots - x_{M}}}}} \right\}$  subject  to  0 ≺ x₁, x₂, …  , x_(M) ≺ 1; 1 < α ≤ M, where theLagrangian multiplier X is used to balance the visual qualities of theresulting 2D and 3D views; coefficient α determines overall intensity ofthe 2D image and it is adjusted in the optimization process to eliminateor suppress artifacts of the 2D image.
 18. The method as claimed inclaim 16 or claim 17 wherein as to a certain stereoscopic image(y_(L),y_(R)), set the m_(L) atom frames of the sequence x₁, x₂, . . . ,x_(M) as a·y_(L) and the m_(R) atom frames of the atom frame sequence asb·y_(R), m_(L)+m_(R)<M, where a and b are positive real values whilem_(L) and m_(R) are positive integers, and choose the remainingM−m_(L)−m_(R) frames as compensation signals, thus generating y₀=x₁+x₂+. . . +x_(M) as a ghost-free two-dimensional version of thethree-dimensional scene.
 19. The method as claimed in claim 16 or claim17 wherein with four sequences of atom frames asx₁=y_(L),x₂=αy₀−y_(L),x₃=y_(R),x₄=αy₀−y_(R), the modulation weightingare respectively w_(L)=(1,0,0,0)

w_(R)=(0,0,1,0), and coefficient α is chosen between [1,4].
 20. Themethod as claimed in claim 15 wherein given a secret image y₁ and acover (e.g., noise or camouflage) image y₀, the perceptual bifurcationeffect for the purpose of security/privacy display can be achieved bycomputing the M atom frames x₁, x₂, . . . , x_(M) and the amplitudemodulation vector w=(w₁, w₂, . . . , w_(M)) to satisfy the followingequation sufficiently${{{well}\; \left( {y_{0},y_{1}} \right)} \approx {\frac{1}{C(M)}\left( {x_{1},x_{2},\ldots \mspace{14mu},x_{M}} \right)\begin{pmatrix}1 & w_{1} \\1 & w_{2} \\\vdots & \vdots \\1 & w_{M}\end{pmatrix}}},$ where C(M) is a constant depending on M, The resultingM atom frames x₁, x₂, . . . , x_(M) are displayed cyclically at highspeed, and they generate cover image y₀ for those without using matchedTPVM viewing devices and at the same time they generate secret image y₁for the authorized user(s) who is equipped with a matched, possiblyencrypted viewing device.
 21. The method as claimed in claim 20 whereinthe said M atom frames x₁, x₂, . . . , x_(M) including two atom frameswhich are used to generating a secrete image y₁ and a cover image y₀,while x₁≈y₁, y₀=x₁+x₂.
 22. The method as claimed in claim 21 wherein thesaid cover image y₀ is set as a random noise n, and set the atom framex₂=n−x₁, thus the secrete image y₁=x₁ is covered by random noisey₀=x₁+x₂=n.